Sequential Multiple Hypothesis Testing with Type I Error Control

This work studies multiple hypothesis testing in the setting when we obtain data sequentially and may choose when to stop sampling. We summarize the notion of a sequential pvalue (one that can be continually updated and still maintain a type I error guarantee) and provide several examples from the literature. This tool allows us to convert fixedhorizon step-up or step-down multiple hypothesis testing procedures (which includes Benjamini-Hochberg, Holm, and Bonferroni) into a sequential version that allows the statistician to reject a hypothesis as soon as the sequential p-value reaches a threshold while maintaining type I error control. We show that if the original procedure has a type I error guarantee in a certain family (including FDR and FWER), then the sequential conversion inherits an analogous guarantee. The conversion also allows for allocating samples in a data-dependent way, and we provide simulated experiments demonstrating an increased number of rejections when compared to the fixed-horizon setting.